{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "机器学习kaggle案例：沃尔玛招聘 - 商店销售预测.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/824024445/KaggleCases/blob/master/Walmart%20Recruiting%20-%20Store%20Sales%20Forecasting.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRD6oFhg_7HT",
        "colab_type": "text"
      },
      "source": [
        "## 一、简介"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rfuPRSYAS0l_",
        "colab_type": "text"
      },
      "source": [
        "### 1.1 比赛描述\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDLU6Fqn_52w",
        "colab_type": "text"
      },
      "source": [
        "建模零售数据的一个挑战是需要根据有限的历史做出决策。如果圣诞节一年一次，那么有机会看到战略决策如何影响到底线。\n",
        "\n",
        "\n",
        "在此招聘竞赛中，为求职者提供位于不同地区的45家沃尔玛商店的历史销售数据。每个商店都包含许多部门，参与者必须为每个商店中的每个部门预测销售额。要添加挑战，选定的假日降价事件将包含在数据集中。众所周知，这些降价会影响销售，但预测哪些部门受到影响以及影响程度具有挑战性。\n",
        "\n",
        "想要在世界上最大的一些数据集的良好环境中工作吗？这是向沃尔玛招聘团队展示您的模特气概的机会。\n",
        "\n",
        "这项比赛计入排名和成就。  如果您希望考虑参加沃尔玛的面试，请在第一次参加时选中“允许主持人与我联系”复选框。 \n",
        "\n",
        "你必须在招募比赛中作为个人参加比赛。您只能使用提供的数据进行预测。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lzuxgtGeS2u5",
        "colab_type": "text"
      },
      "source": [
        "### 1.2 比赛评估"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T5iqem4GAJJ_",
        "colab_type": "text"
      },
      "source": [
        "本次比赛的加权平均绝对误差（WMAE）评估：\n",
        "\n",
        "![替代文字](https://raw.githubusercontent.com/824024445/KaggleCases/master/img/walmart-recruiting-store-sales-forecasting/1-1.jpg)\n",
        "- n是行数\n",
        "- yi是真实销售额\n",
        "- wi是权重，如果该周是假日周，wi=5，否则为1\n",
        "\n",
        "提交文件：Id列是通过将Store，Dept和Date与下划线连接而形成的（例如Store_Dept_2012-11-02）\n",
        "\n",
        "对于测试集中的每一行（商店+部门+日期三元组），您应该预测该部门的每周销售额。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9g0-pTNS5VW",
        "colab_type": "text"
      },
      "source": [
        "### 1.3 数据描述"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OoGSB4_3DHNV",
        "colab_type": "text"
      },
      "source": [
        "您将获得位于不同地区的45家沃尔玛商店的历史销售数据。每个商店都包含许多部门，您的任务是预测每个商店的部门范围内的销售额。\n",
        "\n",
        "此外，沃尔玛全年举办多项促销降价活动。这些降价活动在突出的假期之前，其中最大的四个是超级碗，劳动节，感恩节和圣诞节。包括这些假期的周数在评估中的加权比非假日周高五倍。本次比赛提出的部分挑战是在没有完整/理想的历史数据的情况下模拟降价对这些假期周的影响。\n",
        "\n",
        "**stores.csv:**  \n",
        "此文件包含有关45个商店的匿名信息，指示商店的类型和大小。\n",
        "\n",
        "**train.csv:**  \n",
        "这是历史销售数据，涵盖2010-02-05至2012-11-01。在此文件中，您将找到以下字段：  \n",
        "Store - 商店编号  \n",
        "Dept - 部门编号  \n",
        "Date - 一周  \n",
        "Weekly_Sales - 给定商店中给定部门的销售额(目标值)  \n",
        "sHoliday - 周是否是一个特殊的假日周  \n",
        "\n",
        "**test.csv:**  \n",
        "此文件与train.csv相同，但我们保留了每周销售额。您必须预测此文件中每个商店，部门和日期三元组的销售额。\n",
        "\n",
        "**features.csv:**\n",
        "此文件包含与给定日期的商店，部门和区域活动相关的其他数据。它包含以下字段：  \n",
        "Store - 商店编号  \n",
        "Date - 一周  \n",
        "Temperature - 该地区的平均温度  \n",
        "Fuel_Price - 该地区的燃料成本  \n",
        "MarkDown1-5 - 与沃尔玛正在运营的促销降价相关的匿名数据。MarkDown数据仅在2011年11月之后提供，并非始终适用于所有商店。任何缺失值都标有NA。  \n",
        "CPI - 消费者物价指数  \n",
        "Unemployment - 失业率  \n",
        "IsHoliday - 周是否是一个特殊的假日周  \n",
        "\n",
        "\n",
        "为方便起见，数据集中的四个假期在接下来的几周内（并非所有假期都在数据中）：\n",
        "\n",
        "\n",
        "超级碗：2月12日至10日，11月2日至11日，10月2日至12日，2月8日至2月13  \n",
        "日劳动节：10月9日至10日，9月9日至9日，9月9日至9月12日-13  \n",
        "感恩节：26-Nov- 10,25 -Nov-11,23-Nov-12,29-Nov-13  \n",
        "圣诞节：31-Dec-10,30-Dec-11,28-Dec-12,27-Dec -13  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "peeFvAaMS8SB",
        "colab_type": "text"
      },
      "source": [
        "## 二、代码"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JY1lBSOsS_Xk",
        "colab_type": "text"
      },
      "source": [
        "### 2.1 获取数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U2yzsz3uTqZx",
        "colab_type": "text"
      },
      "source": [
        "#### 2.1.1 下载数据\n",
        "我写了一个小函数来实现数据的下载，数据全都是官网原版数据，我存到了我的github上。（https://github.com/824024445/KaggleCases）  \n",
        "\n",
        "所有数据都下载到了你当前文件夹下的datasets文件下，每个案例涉及到的数据全部下载到了以该案例命名的文件夹下。  \n",
        "\n",
        "我所有的kaggle案例的博客，下载数据均会使用这个函数，只需要修改前两个常量即可。\n",
        "> 注：此函数只用于下载数据，函数在该代码框内就运行了。不再用到其它代码中，包括常量，也不会用在其他地方。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ddwS2eXCTupr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import zipfile\n",
        "from six.moves import urllib\n",
        "\n",
        "FILE_NAME = \"walmart-recruiting-store-sales-forecasting.zip\" #文件名\n",
        "DATA_PATH =\"datasets/walmart-recruiting-store-sales-forecasting\" #存储文件的文件夹，取跟文件相同（相近）的名字便于区分\n",
        "DATA_URL = \"https://github.com/824024445/KaggleCases/blob/master/datasets/\" + FILE_NAME + \"?raw=true\"\n",
        "\n",
        "\n",
        "def fetch_data(data_url=DATA_URL, data_path=DATA_PATH, file_name=FILE_NAME):\n",
        "    if not os.path.isdir(data_path): #查看当前文件夹下是否存在\"datasets/titanic\"，没有的话创建\n",
        "        os.makedirs(data_path)\n",
        "    zip_path = os.path.join(data_path, file_name) #下载到本地的文件的路径及名称\n",
        "    # urlretrieve()方法直接将远程数据下载到本地\n",
        "    urllib.request.urlretrieve(data_url, zip_path) #第二个参数zip_path是保存到的本地路径\n",
        "    data_zip = zipfile.ZipFile(zip_path)\n",
        "    data_zip.extractall(path=data_path) #什么参数都不输入就是默认解压到当前文件,为了保持统一，是泰坦尼克的数据就全部存到titanic文件夹下\n",
        "    data_zip.close()\n",
        "fetch_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_WTk80ApTrAN",
        "colab_type": "text"
      },
      "source": [
        "#### 2.1.2 读取数据"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D2LShOK2T2XO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "5f2e6341-e297-498f-f218-f70bfc4246b2"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "train_df = pd.read_csv(\"datasets/walmart-recruiting-store-sales-forecasting/train.csv\")\n",
        "test_df = pd.read_csv(\"datasets/walmart-recruiting-store-sales-forecasting/test.csv\")\n",
        "features = pd.read_csv(\"datasets/walmart-recruiting-store-sales-forecasting/features.csv\")\n",
        "stores = pd.read_csv(\"datasets/walmart-recruiting-store-sales-forecasting/stores.csv\")\n",
        "\n",
        "train_df = train_df.merge(features, on=[\"Store\", \"Date\"], how=\"left\").merge(stores, on=\"Store\", how=\"left\")\n",
        "test_df = test_df.merge(features, on=[\"Store\", \"Date\"], how=\"left\").merge(stores, on=\"Store\", how=\"left\")\n",
        "combine = [train_df, test_df]\n",
        "train_df.head()\n"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>Dept</th>\n",
              "      <th>Date</th>\n",
              "      <th>Weekly_Sales</th>\n",
              "      <th>IsHoliday_x</th>\n",
              "      <th>Temperature</th>\n",
              "      <th>Fuel_Price</th>\n",
              "      <th>MarkDown1</th>\n",
              "      <th>MarkDown2</th>\n",
              "      <th>MarkDown3</th>\n",
              "      <th>MarkDown4</th>\n",
              "      <th>MarkDown5</th>\n",
              "      <th>CPI</th>\n",
              "      <th>Unemployment</th>\n",
              "      <th>IsHoliday_y</th>\n",
              "      <th>Type</th>\n",
              "      <th>Size</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-05</td>\n",
              "      <td>24924.50</td>\n",
              "      <td>False</td>\n",
              "      <td>42.31</td>\n",
              "      <td>2.572</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.096358</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>A</td>\n",
              "      <td>151315</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-12</td>\n",
              "      <td>46039.49</td>\n",
              "      <td>True</td>\n",
              "      <td>38.51</td>\n",
              "      <td>2.548</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.242170</td>\n",
              "      <td>8.106</td>\n",
              "      <td>True</td>\n",
              "      <td>A</td>\n",
              "      <td>151315</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-19</td>\n",
              "      <td>41595.55</td>\n",
              "      <td>False</td>\n",
              "      <td>39.93</td>\n",
              "      <td>2.514</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.289143</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>A</td>\n",
              "      <td>151315</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-26</td>\n",
              "      <td>19403.54</td>\n",
              "      <td>False</td>\n",
              "      <td>46.63</td>\n",
              "      <td>2.561</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.319643</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>A</td>\n",
              "      <td>151315</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-03-05</td>\n",
              "      <td>21827.90</td>\n",
              "      <td>False</td>\n",
              "      <td>46.50</td>\n",
              "      <td>2.625</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.350143</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>A</td>\n",
              "      <td>151315</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Store  Dept        Date  ...  IsHoliday_y  Type    Size\n",
              "0      1     1  2010-02-05  ...        False     A  151315\n",
              "1      1     1  2010-02-12  ...         True     A  151315\n",
              "2      1     1  2010-02-19  ...        False     A  151315\n",
              "3      1     1  2010-02-26  ...        False     A  151315\n",
              "4      1     1  2010-03-05  ...        False     A  151315\n",
              "\n",
              "[5 rows x 17 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j6PEUuz0NI2S",
        "colab_type": "text"
      },
      "source": [
        "### 2.2 初步观察数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Giyb4BY0NOAQ",
        "colab_type": "text"
      },
      "source": [
        "#### 2.2.1 info()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VaS8X7MwNNdU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "6aa1d023-5682-4cbe-c376-4018c9db7d4c"
      },
      "source": [
        "train_df.info()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 421570 entries, 0 to 421569\n",
            "Data columns (total 17 columns):\n",
            "Store           421570 non-null int64\n",
            "Dept            421570 non-null int64\n",
            "Date            421570 non-null object\n",
            "Weekly_Sales    421570 non-null float64\n",
            "IsHoliday_x     421570 non-null bool\n",
            "Temperature     421570 non-null float64\n",
            "Fuel_Price      421570 non-null float64\n",
            "MarkDown1       150681 non-null float64\n",
            "MarkDown2       111248 non-null float64\n",
            "MarkDown3       137091 non-null float64\n",
            "MarkDown4       134967 non-null float64\n",
            "MarkDown5       151432 non-null float64\n",
            "CPI             421570 non-null float64\n",
            "Unemployment    421570 non-null float64\n",
            "IsHoliday_y     421570 non-null bool\n",
            "Type            421570 non-null object\n",
            "Size            421570 non-null int64\n",
            "dtypes: bool(2), float64(10), int64(3), object(2)\n",
            "memory usage: 52.3+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rbsVmjNYOiKz",
        "colab_type": "text"
      },
      "source": [
        "观察到：\n",
        "- MarkDown有太多缺失值，但是后面查看test发现test该特征比较完整，且后面查看想关性，该特征有挺高的相关性\n",
        "- 其余特征没有空值，等会就不用补充缺失值了"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wm7hZYUsQZAd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "9c568c47-6856-4ccd-e162-e4a83f77a088"
      },
      "source": [
        "test_df.info()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 115064 entries, 0 to 115063\n",
            "Data columns (total 16 columns):\n",
            "Store           115064 non-null int64\n",
            "Dept            115064 non-null int64\n",
            "Date            115064 non-null object\n",
            "IsHoliday_x     115064 non-null bool\n",
            "Temperature     115064 non-null float64\n",
            "Fuel_Price      115064 non-null float64\n",
            "MarkDown1       114915 non-null float64\n",
            "MarkDown2       86437 non-null float64\n",
            "MarkDown3       105235 non-null float64\n",
            "MarkDown4       102176 non-null float64\n",
            "MarkDown5       115064 non-null float64\n",
            "CPI             76902 non-null float64\n",
            "Unemployment    76902 non-null float64\n",
            "IsHoliday_y     115064 non-null bool\n",
            "Type            115064 non-null object\n",
            "Size            115064 non-null int64\n",
            "dtypes: bool(2), float64(9), int64(3), object(2)\n",
            "memory usage: 13.4+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHsQf5x6Qho_",
        "colab_type": "text"
      },
      "source": [
        "观察测试集发现：\n",
        "- 测试集的markdown数据还挺全的，这个特征可能还是有用的。就先做去掉这个特征的，然后提升阶段再考虑怎么利用这部分数据吧。\n",
        "- 测试集的CPI和Unemployment有空值"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLgMsAg4QSBH",
        "colab_type": "text"
      },
      "source": [
        "#### 2.2.2 describe()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGJtAG-wQXLO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "56984687-04fb-4f87-f879-f26684e042a5"
      },
      "source": [
        "train_df.describe()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>Dept</th>\n",
              "      <th>Weekly_Sales</th>\n",
              "      <th>Temperature</th>\n",
              "      <th>Fuel_Price</th>\n",
              "      <th>MarkDown1</th>\n",
              "      <th>MarkDown2</th>\n",
              "      <th>MarkDown3</th>\n",
              "      <th>MarkDown4</th>\n",
              "      <th>MarkDown5</th>\n",
              "      <th>CPI</th>\n",
              "      <th>Unemployment</th>\n",
              "      <th>Size</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>150681.000000</td>\n",
              "      <td>111248.000000</td>\n",
              "      <td>137091.000000</td>\n",
              "      <td>134967.000000</td>\n",
              "      <td>151432.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "      <td>421570.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>22.200546</td>\n",
              "      <td>44.260317</td>\n",
              "      <td>15981.258123</td>\n",
              "      <td>60.090059</td>\n",
              "      <td>3.361027</td>\n",
              "      <td>7246.420196</td>\n",
              "      <td>3334.628621</td>\n",
              "      <td>1439.421384</td>\n",
              "      <td>3383.168256</td>\n",
              "      <td>4628.975079</td>\n",
              "      <td>171.201947</td>\n",
              "      <td>7.960289</td>\n",
              "      <td>136727.915739</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>12.785297</td>\n",
              "      <td>30.492054</td>\n",
              "      <td>22711.183519</td>\n",
              "      <td>18.447931</td>\n",
              "      <td>0.458515</td>\n",
              "      <td>8291.221345</td>\n",
              "      <td>9475.357325</td>\n",
              "      <td>9623.078290</td>\n",
              "      <td>6292.384031</td>\n",
              "      <td>5962.887455</td>\n",
              "      <td>39.159276</td>\n",
              "      <td>1.863296</td>\n",
              "      <td>60980.583328</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-4988.940000</td>\n",
              "      <td>-2.060000</td>\n",
              "      <td>2.472000</td>\n",
              "      <td>0.270000</td>\n",
              "      <td>-265.760000</td>\n",
              "      <td>-29.100000</td>\n",
              "      <td>0.220000</td>\n",
              "      <td>135.160000</td>\n",
              "      <td>126.064000</td>\n",
              "      <td>3.879000</td>\n",
              "      <td>34875.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>11.000000</td>\n",
              "      <td>18.000000</td>\n",
              "      <td>2079.650000</td>\n",
              "      <td>46.680000</td>\n",
              "      <td>2.933000</td>\n",
              "      <td>2240.270000</td>\n",
              "      <td>41.600000</td>\n",
              "      <td>5.080000</td>\n",
              "      <td>504.220000</td>\n",
              "      <td>1878.440000</td>\n",
              "      <td>132.022667</td>\n",
              "      <td>6.891000</td>\n",
              "      <td>93638.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>22.000000</td>\n",
              "      <td>37.000000</td>\n",
              "      <td>7612.030000</td>\n",
              "      <td>62.090000</td>\n",
              "      <td>3.452000</td>\n",
              "      <td>5347.450000</td>\n",
              "      <td>192.000000</td>\n",
              "      <td>24.600000</td>\n",
              "      <td>1481.310000</td>\n",
              "      <td>3359.450000</td>\n",
              "      <td>182.318780</td>\n",
              "      <td>7.866000</td>\n",
              "      <td>140167.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>33.000000</td>\n",
              "      <td>74.000000</td>\n",
              "      <td>20205.852500</td>\n",
              "      <td>74.280000</td>\n",
              "      <td>3.738000</td>\n",
              "      <td>9210.900000</td>\n",
              "      <td>1926.940000</td>\n",
              "      <td>103.990000</td>\n",
              "      <td>3595.040000</td>\n",
              "      <td>5563.800000</td>\n",
              "      <td>212.416993</td>\n",
              "      <td>8.572000</td>\n",
              "      <td>202505.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>45.000000</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>693099.360000</td>\n",
              "      <td>100.140000</td>\n",
              "      <td>4.468000</td>\n",
              "      <td>88646.760000</td>\n",
              "      <td>104519.540000</td>\n",
              "      <td>141630.610000</td>\n",
              "      <td>67474.850000</td>\n",
              "      <td>108519.280000</td>\n",
              "      <td>227.232807</td>\n",
              "      <td>14.313000</td>\n",
              "      <td>219622.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Store           Dept  ...   Unemployment           Size\n",
              "count  421570.000000  421570.000000  ...  421570.000000  421570.000000\n",
              "mean       22.200546      44.260317  ...       7.960289  136727.915739\n",
              "std        12.785297      30.492054  ...       1.863296   60980.583328\n",
              "min         1.000000       1.000000  ...       3.879000   34875.000000\n",
              "25%        11.000000      18.000000  ...       6.891000   93638.000000\n",
              "50%        22.000000      37.000000  ...       7.866000  140167.000000\n",
              "75%        33.000000      74.000000  ...       8.572000  202505.000000\n",
              "max        45.000000      99.000000  ...      14.313000  219622.000000\n",
              "\n",
              "[8 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJaEFc87Q961",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "2ed67796-1904-4cbd-c48e-2150885aa6a8"
      },
      "source": [
        "train_df.describe(include=\"O\")"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Type</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>421570</td>\n",
              "      <td>421570</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>143</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>2011-12-23</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>3027</td>\n",
              "      <td>215478</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Date    Type\n",
              "count       421570  421570\n",
              "unique         143       3\n",
              "top     2011-12-23       A\n",
              "freq          3027  215478"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hIwEYFpSMJL",
        "colab_type": "text"
      },
      "source": [
        "测试集CPI和Unemployment有缺失值，看一下它的结构"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VGiOfb4Ozm0d",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "cff96b78-54e4-413d-b207-7ca00a4fcda5"
      },
      "source": [
        "test_df[[\"CPI\",\"Unemployment\"]].describe()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CPI</th>\n",
              "      <th>Unemployment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>76902.000000</td>\n",
              "      <td>76902.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>176.961347</td>\n",
              "      <td>6.868733</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>41.239967</td>\n",
              "      <td>1.583427</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>131.236226</td>\n",
              "      <td>3.684000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>138.402033</td>\n",
              "      <td>5.771000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>192.304445</td>\n",
              "      <td>6.806000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>223.244532</td>\n",
              "      <td>8.036000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>228.976456</td>\n",
              "      <td>10.199000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                CPI  Unemployment\n",
              "count  76902.000000  76902.000000\n",
              "mean     176.961347      6.868733\n",
              "std       41.239967      1.583427\n",
              "min      131.236226      3.684000\n",
              "25%      138.402033      5.771000\n",
              "50%      192.304445      6.806000\n",
              "75%      223.244532      8.036000\n",
              "max      228.976456     10.199000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FS8r5RpgTb4T",
        "colab_type": "text"
      },
      "source": [
        "#### 2.2.3 corr()"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RguzCg9gTfrY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "cef5d83d-445e-485f-bf5d-5236094f127d"
      },
      "source": [
        "corr_matrix = train_df.corr()\n",
        "corr_matrix.Weekly_Sales.sort_values(ascending=False)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Weekly_Sales    1.000000\n",
              "Size            0.243828\n",
              "Dept            0.148032\n",
              "MarkDown5       0.090362\n",
              "MarkDown1       0.085251\n",
              "MarkDown3       0.060385\n",
              "MarkDown4       0.045414\n",
              "MarkDown2       0.024130\n",
              "IsHoliday_y     0.012774\n",
              "IsHoliday_x     0.012774\n",
              "Fuel_Price     -0.000120\n",
              "Temperature    -0.002312\n",
              "CPI            -0.020921\n",
              "Unemployment   -0.025864\n",
              "Store          -0.085195\n",
              "Name: Weekly_Sales, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oCRPNgX_XYXv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "a18436c9-5c8a-4f88-ee99-762ca1dd39f4"
      },
      "source": [
        "corr_matrix[[\"MarkDown1\",\"MarkDown2\",\"MarkDown3\",\"MarkDown4\",\"MarkDown5\"]].sort_values(by=\"MarkDown5\", ascending=False)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MarkDown1</th>\n",
              "      <th>MarkDown2</th>\n",
              "      <th>MarkDown3</th>\n",
              "      <th>MarkDown4</th>\n",
              "      <th>MarkDown5</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>MarkDown5</th>\n",
              "      <td>0.160257</td>\n",
              "      <td>-0.007440</td>\n",
              "      <td>-0.026467</td>\n",
              "      <td>0.107792</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Size</th>\n",
              "      <td>0.345673</td>\n",
              "      <td>0.108827</td>\n",
              "      <td>0.048913</td>\n",
              "      <td>0.168196</td>\n",
              "      <td>0.304575</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>MarkDown1</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.024486</td>\n",
              "      <td>-0.108115</td>\n",
              "      <td>0.819238</td>\n",
              "      <td>0.160257</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>MarkDown4</th>\n",
              "      <td>0.819238</td>\n",
              "      <td>-0.007768</td>\n",
              "      <td>-0.071095</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.107792</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Weekly_Sales</th>\n",
              "      <td>0.085251</td>\n",
              "      <td>0.024130</td>\n",
              "      <td>0.060385</td>\n",
              "      <td>0.045414</td>\n",
              "      <td>0.090362</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CPI</th>\n",
              "      <td>-0.055558</td>\n",
              "      <td>-0.039534</td>\n",
              "      <td>-0.023590</td>\n",
              "      <td>-0.049628</td>\n",
              "      <td>0.060630</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Dept</th>\n",
              "      <td>-0.002426</td>\n",
              "      <td>0.000290</td>\n",
              "      <td>0.001784</td>\n",
              "      <td>0.004257</td>\n",
              "      <td>0.000109</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Unemployment</th>\n",
              "      <td>0.050285</td>\n",
              "      <td>0.020940</td>\n",
              "      <td>0.012818</td>\n",
              "      <td>0.024963</td>\n",
              "      <td>-0.003843</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>MarkDown2</th>\n",
              "      <td>0.024486</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.050108</td>\n",
              "      <td>-0.007768</td>\n",
              "      <td>-0.007440</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Temperature</th>\n",
              "      <td>-0.040594</td>\n",
              "      <td>-0.323927</td>\n",
              "      <td>-0.096880</td>\n",
              "      <td>-0.063947</td>\n",
              "      <td>-0.017544</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>MarkDown3</th>\n",
              "      <td>-0.108115</td>\n",
              "      <td>-0.050108</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-0.071095</td>\n",
              "      <td>-0.026467</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Store</th>\n",
              "      <td>-0.119588</td>\n",
              "      <td>-0.035173</td>\n",
              "      <td>-0.031556</td>\n",
              "      <td>-0.009941</td>\n",
              "      <td>-0.026634</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>IsHoliday_x</th>\n",
              "      <td>-0.035586</td>\n",
              "      <td>0.334818</td>\n",
              "      <td>0.427960</td>\n",
              "      <td>-0.000562</td>\n",
              "      <td>-0.053719</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>IsHoliday_y</th>\n",
              "      <td>-0.035586</td>\n",
              "      <td>0.334818</td>\n",
              "      <td>0.427960</td>\n",
              "      <td>-0.000562</td>\n",
              "      <td>-0.053719</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Fuel_Price</th>\n",
              "      <td>0.061371</td>\n",
              "      <td>-0.220895</td>\n",
              "      <td>-0.102092</td>\n",
              "      <td>-0.044986</td>\n",
              "      <td>-0.128065</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              MarkDown1  MarkDown2  MarkDown3  MarkDown4  MarkDown5\n",
              "MarkDown5      0.160257  -0.007440  -0.026467   0.107792   1.000000\n",
              "Size           0.345673   0.108827   0.048913   0.168196   0.304575\n",
              "MarkDown1      1.000000   0.024486  -0.108115   0.819238   0.160257\n",
              "MarkDown4      0.819238  -0.007768  -0.071095   1.000000   0.107792\n",
              "Weekly_Sales   0.085251   0.024130   0.060385   0.045414   0.090362\n",
              "CPI           -0.055558  -0.039534  -0.023590  -0.049628   0.060630\n",
              "Dept          -0.002426   0.000290   0.001784   0.004257   0.000109\n",
              "Unemployment   0.050285   0.020940   0.012818   0.024963  -0.003843\n",
              "MarkDown2      0.024486   1.000000  -0.050108  -0.007768  -0.007440\n",
              "Temperature   -0.040594  -0.323927  -0.096880  -0.063947  -0.017544\n",
              "MarkDown3     -0.108115  -0.050108   1.000000  -0.071095  -0.026467\n",
              "Store         -0.119588  -0.035173  -0.031556  -0.009941  -0.026634\n",
              "IsHoliday_x   -0.035586   0.334818   0.427960  -0.000562  -0.053719\n",
              "IsHoliday_y   -0.035586   0.334818   0.427960  -0.000562  -0.053719\n",
              "Fuel_Price     0.061371  -0.220895  -0.102092  -0.044986  -0.128065"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PcJONVjUX4NI",
        "colab_type": "text"
      },
      "source": [
        "- markdown1和4的关联度比较大，只需要要一个就行，删除markdown4"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQtFMkvpUTBz",
        "colab_type": "text"
      },
      "source": [
        "### 2.3 数据清洗"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cnDbE0imY6Hm",
        "colab_type": "text"
      },
      "source": [
        "#### 2.3.1 缺失值处理\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pjn3AyDDt3P2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Markdown 对于训练集markdown的缺失，这里先不处理，等会分成两个数据集，一个含缺失markdown然后填充，一个去掉这些数据\n",
        "test_df[['MarkDown1','MarkDown2','MarkDown3','MarkDown5']] = test_df[['MarkDown1','MarkDown2','MarkDown3','MarkDown5']].fillna(0)\n",
        "test_df[[\"CPI\",\"Unemployment\"]] = test_df[[\"CPI\",\"Unemployment\"]].fillna(method=\"ffill\")                                                                                                                                                                                            "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LKc25-l6Lrws",
        "colab_type": "text"
      },
      "source": [
        "#### 2.3.2 创建新特征\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3DCLPzZ6P8Dt",
        "colab_type": "text"
      },
      "source": [
        "type转变成onehot编码"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jAebJFsmPeyR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "f7ab9a76-c501-49ce-a5b7-7de1f4704f46"
      },
      "source": [
        "train_df = pd.get_dummies(train_df, columns=[\"Type\"])\n",
        "test_df = pd.get_dummies(test_df, columns=[\"Type\"])\n",
        "train_df.head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>Dept</th>\n",
              "      <th>Date</th>\n",
              "      <th>Weekly_Sales</th>\n",
              "      <th>IsHoliday_x</th>\n",
              "      <th>Temperature</th>\n",
              "      <th>Fuel_Price</th>\n",
              "      <th>MarkDown1</th>\n",
              "      <th>MarkDown2</th>\n",
              "      <th>MarkDown3</th>\n",
              "      <th>MarkDown4</th>\n",
              "      <th>MarkDown5</th>\n",
              "      <th>CPI</th>\n",
              "      <th>Unemployment</th>\n",
              "      <th>IsHoliday_y</th>\n",
              "      <th>Size</th>\n",
              "      <th>Type_A</th>\n",
              "      <th>Type_B</th>\n",
              "      <th>Type_C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-05</td>\n",
              "      <td>24924.50</td>\n",
              "      <td>False</td>\n",
              "      <td>42.31</td>\n",
              "      <td>2.572</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.096358</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>151315</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-12</td>\n",
              "      <td>46039.49</td>\n",
              "      <td>True</td>\n",
              "      <td>38.51</td>\n",
              "      <td>2.548</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.242170</td>\n",
              "      <td>8.106</td>\n",
              "      <td>True</td>\n",
              "      <td>151315</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-19</td>\n",
              "      <td>41595.55</td>\n",
              "      <td>False</td>\n",
              "      <td>39.93</td>\n",
              "      <td>2.514</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.289143</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>151315</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-02-26</td>\n",
              "      <td>19403.54</td>\n",
              "      <td>False</td>\n",
              "      <td>46.63</td>\n",
              "      <td>2.561</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.319643</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>151315</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2010-03-05</td>\n",
              "      <td>21827.90</td>\n",
              "      <td>False</td>\n",
              "      <td>46.50</td>\n",
              "      <td>2.625</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>211.350143</td>\n",
              "      <td>8.106</td>\n",
              "      <td>False</td>\n",
              "      <td>151315</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Store  Dept        Date  Weekly_Sales  ...    Size  Type_A  Type_B  Type_C\n",
              "0      1     1  2010-02-05      24924.50  ...  151315       1       0       0\n",
              "1      1     1  2010-02-12      46039.49  ...  151315       1       0       0\n",
              "2      1     1  2010-02-19      41595.55  ...  151315       1       0       0\n",
              "3      1     1  2010-02-26      19403.54  ...  151315       1       0       0\n",
              "4      1     1  2010-03-05      21827.90  ...  151315       1       0       0\n",
              "\n",
              "[5 rows x 19 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqwMTVrIP_Xn",
        "colab_type": "text"
      },
      "source": [
        "把日期换成月份"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8F1pL5XpLrXb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_df['Month'] = pd.to_datetime(train_df['Date']).dt.month\n",
        "test_df[\"Month\"] = pd.to_datetime(test_df['Date']).dt.month\n",
        "#等下记得删除Date，test的暂时先不删，后面要用"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5088oKlQBZf",
        "colab_type": "text"
      },
      "source": [
        "温度  \n",
        "想来，人们在极端天气的时候不太会出门。所以把数据分成两组：小于22.01，大于91.03（根据温度分布划分的，画柱状图可得，我已经删掉了）"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dyOECgUIQiGh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "4e84119e-b0f0-4475-db7f-913da8bbcdf2"
      },
      "source": [
        "train_df.loc[(train_df[\"Temperature\"]<22.01)|(train_df[\"Temperature\"]>91.03), \"Is_temp_extr\"]=1\n",
        "train_df.loc[(train_df[\"Temperature\"]>=22.01)& (train_df[\"Temperature\"]<=91.03), \"Is_temp_extr\"]=0\n",
        "\n",
        "test_df.loc[(test_df[\"Temperature\"]<22.01)|(test_df[\"Temperature\"]>91.03), \"Is_temp_extr\"]=1\n",
        "test_df.loc[(test_df[\"Temperature\"]>=22.01)& (test_df[\"Temperature\"]<=91.03), \"Is_temp_extr\"]=0\n",
        "\n",
        "train_df.corr().Weekly_Sales.sort_values(ascending=False)[[\"Temperature\", \"Is_temp_extr\"]]\n",
        "#提取新特征后相关性提升了十多倍 等下记得把这个特征删除。"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Temperature    -0.002312\n",
              "Is_temp_extr   -0.030016\n",
              "Name: Weekly_Sales, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_HQasKbwXwIm",
        "colab_type": "text"
      },
      "source": [
        "燃油价格  \n",
        "人们会因为燃油费太贵不出门吗？"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XdSPvPhbX05C",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "a57c0c0c-c850-41b8-8980-1bb975301e7c"
      },
      "source": [
        "train_df.loc[train_df[\"Fuel_Price\"]>3.47, \"Is_fuel_expen\"]=1\n",
        "train_df.loc[train_df[\"Fuel_Price\"]<=3.47, \"Is_fuel_expen\"]=0\n",
        "#无论怎么改，这个相关性都很低，所以这个特征等下去除\n",
        "train_df.corr().Weekly_Sales.sort_values(ascending=False)[[\"Fuel_Price\", \"Is_fuel_expen\"]]"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Fuel_Price      -0.000120\n",
              "Is_fuel_expen   -0.006626\n",
              "Name: Weekly_Sales, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSX4AypAa4yV",
        "colab_type": "text"
      },
      "source": [
        "IsHoliday  \n",
        "由于前面合并表格的时候的问题，出现了两个isholidy,删掉一个即可。\n",
        "另外，把bool值换成0和5(后面权重）"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ugbCrp8tMVhg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "ab3b242f-b365-4d52-a45e-20af8e9f8bec"
      },
      "source": [
        "train_df[\"IsHoliday\"] = train_df[\"IsHoliday_x\"].replace(True, 5).replace(False,0)\n",
        "test_df[\"IsHoliday\"] = test_df[\"IsHoliday_x\"].replace(True, 5).replace(False,0)\n",
        "\n",
        "train_df.corr().Weekly_Sales.sort_values(ascending=False)[[\"IsHoliday_x\", \"IsHoliday\"]]"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "IsHoliday_x    0.012774\n",
              "IsHoliday      0.012774\n",
              "Name: Weekly_Sales, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFrk4s3YOUKk",
        "colab_type": "text"
      },
      "source": [
        "#### 2.3.3 删除特征"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-yVryLA4OWRb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_df = train_df.drop([\"IsHoliday_x\", \"IsHoliday_y\",'MarkDown4',\"Date\", \"Temperature\", \"Fuel_Price\",\"Is_fuel_expen\"], axis=1)\n",
        "\n",
        "#这是后面提交表格需要用到的变量，用到了测试集的date特征，先在这里给id变量赋值，然后就可以吧date特征删除了\n",
        "id = test_df[\"Store\"].astype(str)+\"_\"+test_df[\"Dept\"].astype(str)+\"_\"+test_df[\"Date\"].astype(str)\n",
        "test_df = test_df.drop([\"IsHoliday_x\", \"IsHoliday_y\", \"MarkDown4\", \"Date\",\"Temperature\", \"Fuel_Price\"], axis=1) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZU9J7uX3c5hp",
        "colab_type": "text"
      },
      "source": [
        "#### 2.3.4 最终检查\n",
        "将数据集用到模型前，一定要确保没有空值，所以最后再检查一下"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GxtSDpBsgN5t",
        "colab_type": "text"
      },
      "source": [
        "先把训练集做成两份：一份含缺失的markdown，一个去除掉这些数据"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1cU3cqfUgLJU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "1019b2da-17bc-4660-9092-29fd362551f4"
      },
      "source": [
        "train_df_one = train_df.copy()\n",
        "train_df_two = train_df.copy()\n",
        "train_df_one[['MarkDown1','MarkDown2','MarkDown3','MarkDown5']] = train_df_one[['MarkDown1','MarkDown2','MarkDown3','MarkDown5']].fillna(0)\n",
        "train_df_two.dropna(inplace=True)\n",
        "\n",
        "train_df_one.info()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 421570 entries, 0 to 421569\n",
            "Data columns (total 16 columns):\n",
            "Store           421570 non-null int64\n",
            "Dept            421570 non-null int64\n",
            "Weekly_Sales    421570 non-null float64\n",
            "MarkDown1       421570 non-null float64\n",
            "MarkDown2       421570 non-null float64\n",
            "MarkDown3       421570 non-null float64\n",
            "MarkDown5       421570 non-null float64\n",
            "CPI             421570 non-null float64\n",
            "Unemployment    421570 non-null float64\n",
            "Size            421570 non-null int64\n",
            "Type_A          421570 non-null uint8\n",
            "Type_B          421570 non-null uint8\n",
            "Type_C          421570 non-null uint8\n",
            "Month           421570 non-null int64\n",
            "Is_temp_extr    421570 non-null float64\n",
            "IsHoliday       421570 non-null float64\n",
            "dtypes: float64(9), int64(4), uint8(3)\n",
            "memory usage: 46.2 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4m1ZMPbg7tk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "e09a807c-ba37-4d97-ce27-392e58fa8350"
      },
      "source": [
        "train_df_two.info()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 101480 entries, 92 to 421569\n",
            "Data columns (total 16 columns):\n",
            "Store           101480 non-null int64\n",
            "Dept            101480 non-null int64\n",
            "Weekly_Sales    101480 non-null float64\n",
            "MarkDown1       101480 non-null float64\n",
            "MarkDown2       101480 non-null float64\n",
            "MarkDown3       101480 non-null float64\n",
            "MarkDown5       101480 non-null float64\n",
            "CPI             101480 non-null float64\n",
            "Unemployment    101480 non-null float64\n",
            "Size            101480 non-null int64\n",
            "Type_A          101480 non-null uint8\n",
            "Type_B          101480 non-null uint8\n",
            "Type_C          101480 non-null uint8\n",
            "Month           101480 non-null int64\n",
            "Is_temp_extr    101480 non-null float64\n",
            "IsHoliday       101480 non-null float64\n",
            "dtypes: float64(9), int64(4), uint8(3)\n",
            "memory usage: 11.1 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v7_ZL_b8dGeT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "5f939130-6827-4e3f-c10c-245d3a426150"
      },
      "source": [
        "test_df.info()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 115064 entries, 0 to 115063\n",
            "Data columns (total 15 columns):\n",
            "Store           115064 non-null int64\n",
            "Dept            115064 non-null int64\n",
            "MarkDown1       115064 non-null float64\n",
            "MarkDown2       115064 non-null float64\n",
            "MarkDown3       115064 non-null float64\n",
            "MarkDown5       115064 non-null float64\n",
            "CPI             115064 non-null float64\n",
            "Unemployment    115064 non-null float64\n",
            "Size            115064 non-null int64\n",
            "Type_A          115064 non-null uint8\n",
            "Type_B          115064 non-null uint8\n",
            "Type_C          115064 non-null uint8\n",
            "Month           115064 non-null int64\n",
            "Is_temp_extr    115064 non-null float64\n",
            "IsHoliday       115064 non-null float64\n",
            "dtypes: float64(8), int64(4), uint8(3)\n",
            "memory usage: 11.7 MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IzrSUaMhYc6P",
        "colab_type": "text"
      },
      "source": [
        "### 2.4 模型和预测"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJxB1VlbYsYa",
        "colab_type": "text"
      },
      "source": [
        "为了快速测试，写了一个类。我写的案例大部分都回用到这个类。不过每次因为性能评测的指标不同，所以需要微改。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UYg_5OgxYgEC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "import os\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "from sklearn.base import clone\n",
        "\n",
        "class Tester():\n",
        "    def __init__(self, target):\n",
        "        self.target = target\n",
        "        self.datasets = {}\n",
        "        self.models = {}\n",
        "        self.scores = {}\n",
        "        self.cache = {} # 我们添加了一个简单的缓存来加快速度\n",
        "\n",
        "    def addDataset(self, name, df):\n",
        "        self.datasets[name] = df.copy()\n",
        "\n",
        "    def addModel(self, name, model):\n",
        "        self.models[name] = model\n",
        "        \n",
        "    def clearModels(self):\n",
        "        self.models = {}\n",
        "\n",
        "    def clearCache(self):\n",
        "        self.cache = {}\n",
        "    \n",
        "    def testModelWithDataset(self, m_name, df_name, sample_len, cv):\n",
        "        if (m_name, df_name, sample_len, cv) in self.cache:\n",
        "            return self.cache[(m_name, df_name, sample_len, cv)]\n",
        "\n",
        "        clf = clone(self.models[m_name])\n",
        "        \n",
        "        if not sample_len: \n",
        "            sample = self.datasets[df_name]\n",
        "        else: sample = self.datasets[df_name].sample(sample_len)\n",
        "            \n",
        "        X = sample.drop([self.target], axis=1)\n",
        "        Y = sample[self.target]\n",
        "\n",
        "        #评分标准不一样的话，修改这里\n",
        "        weights = X[\"IsHoliday\"]\n",
        "        clf.fit(X, Y)\n",
        "        Y_pred = clf.predict(X)\n",
        "        s = mean_absolute_error(Y, Y_pred, sample_weight=weights)\n",
        "        self.cache[(m_name, df_name, sample_len, cv)] = s\n",
        "\n",
        "        return s\n",
        "\n",
        "    def runTests(self, sample_len=97056, cv=3):\n",
        "        # 在所有添加的数据集上测试添加的模型\n",
        "        for m_name in self.models:\n",
        "            for df_name in self.datasets:\n",
        "                # print('Testing %s' % str((m_name, df_name)), end='')\n",
        "                start = time.time()\n",
        "\n",
        "                score = self.testModelWithDataset(m_name, df_name, sample_len, cv)\n",
        "                self.scores[(m_name, df_name)] = score\n",
        "                \n",
        "                end = time.time()\n",
        "                \n",
        "                # print(' -- %0.2fs ' % (end - start))\n",
        "\n",
        "        print('--- Top 10 Results ---')\n",
        "        # 评分标准改了之后这里也得改\n",
        "        for score in sorted(self.scores.items(), key=lambda x: x[1])[:10]:\n",
        "            # score = int(score[1])\n",
        "            print(score)\n",
        "\n",
        "    def obtian_result(self, X_test):\n",
        "        clf = self.models[sorted(self.scores.items(), key=lambda x: x[1])[0][0]]\n",
        "        Y_pred = clf.predict(X_test)\n",
        "        return Y_pred\n",
        "\n",
        "   "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8nm2udcKi6YT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 165
        },
        "outputId": "e02392c7-f0e1-4732-e313-48ac4a68ff76"
      },
      "source": [
        "from sklearn.ensemble import ExtraTreesRegressor, RandomForestRegressor, GradientBoostingRegressor\n",
        "from sklearn.neighbors import KNeighborsRegressor\n",
        "from sklearn.svm import SVR\n",
        "from sklearn.feature_selection import RFE\n",
        "from sklearn.neural_network import MLPRegressor\n",
        "\n",
        "# 我们将在所有模型中使用测试对象\n",
        "tester = Tester('Weekly_Sales')\n",
        "\n",
        "# 添加数据集\n",
        "tester.addDataset('all_markdown', train_df_one)\n",
        "tester.addDataset('wipe_markdown', train_df_two)\n",
        "\n",
        "# 添加模型\n",
        "knn_reg = KNeighborsRegressor(n_neighbors=10)\n",
        "tree_reg = ExtraTreesRegressor(n_estimators=100,max_features='auto', verbose=1, n_jobs=1)\n",
        "rf_reg = RandomForestRegressor(n_estimators=100,max_features='log2', verbose=1)\n",
        "svr_reg = SVR(kernel='rbf', gamma='auto')\n",
        "mlp_reg = MLPRegressor(hidden_layer_sizes=(10,),  activation='relu', verbose=3)\n",
        "gbrt_reg = GradientBoostingRegressor(max_depth=8, warm_start=True)\n",
        "tester.addModel('KNeighborsRegressor', knn_reg)\n",
        "tester.addModel('ExtraTreesRegressor', tree_reg)\n",
        "tester.addModel('RandomForestRegressor', rf_reg)\n",
        "tester.addModel('SVR', svr_reg)\n",
        "tester.addModel('MLPRegressor', mlp_reg)\n",
        "tester.addModel('GradientBoostingRegressor', gbrt_reg)\n",
        "\n",
        "# 测试\n",
        "tester.runTests()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
            "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:   26.4s finished\n",
            "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
            "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    3.8s finished\n",
            "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
            "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:   28.6s finished\n",
            "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
            "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    3.8s finished\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "15zDQVnLbp3K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = train_df_one.drop([\"Weekly_Sales\"], axis=1)\n",
        "Y = train_df_one[\"Weekly_Sales\"]\n",
        "\n",
        "gbrt_reg.fit(X, Y)\n",
        "Y_pred = gbrt_reg.predict(test_df)\n",
        "submission = pd.DataFrame({\n",
        "        \"Id\": id,\n",
        "        \"Weekly_Sales\": pd.DataFrame(Y_pred)[0]\n",
        "    })\n",
        "id\n",
        "\n",
        "submission.to_csv('submission.csv', index=False)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}